Structure or Noise?

One recurring challenge in many areas of modern physics which analyze complex and vast data sets is to use automated methods to distinguish between structure and noise in the data. A particular challenge is to ensure that these methods are based on some well-understood principles, rather than on an ad hoc procedure. In this paper we discuss how a mechanism for automated theory building that naturally delineates regularity from randomness is provided by a branch of information theory known as rate-distortion theory. In building a model, one usually summarizes the given data in some meaningful way by discarding irrelevant information, so that the modeling process itself can be interpreted as a lossy compression. Model variables should then, as much as possible, render the future and the past conditionally independent. We start from this simple idea, which is also known as causal-shielding, and construct an objective function for model making whose extrema embody the trade-off between a model’s structural complexity and its predictive power. The solutions correspond to a hierarchy of models that, at each level of complexity, achieve optimal predictive power at minimal cost. In the limit of maximal prediction the resulting optimal model identifies a process’s intrinsic organization by extracting the process’s underlying causal states. In this limit, the model’s complexity is given by the statistical complexity, which is known to be minimal for achieving maximum prediction. We introduce the notion of causal compressibility and show in examples how theory building can profit from analyzing a process’s causal compressibility—the process’s characteristic for optimally balancing structure and noise at different levels of representational detail. PACS numbers: 02.50.-r 89.70.+c 05.45.Tp 02.50.Ey Structure or Noise? 2